Image fusion for interventional guidance

ABSTRACT

A method for real-time fusion of a 2D cardiac ultrasound image with a 2D cardiac fluoroscopic image includes acquiring real time synchronized US and fluoroscopic images, detecting a surface contour of an aortic valve in the 2D cardiac ultrasound (US) image relative to an US probe, detecting a pose of the US probe in the 2D cardiac fluoroscopic image, and using pose parameters of the US probe to transform the surface contour of the aortic valve from the 2D cardiac US image to the 2D cardiac fluoroscopic image.

CROSS REFERENCE TO RELATED UNITED STATES APPLICATIONS

This application claims priority from “Real-Time TAVI Navigation: FusingAnatomy from 2D US with Fluoroscopy”, U.S. Provisional Application No.61/602,107 of Mountney, et al., filed Feb. 23, 2012, “Robust Model-basedFusion of Pre- and Intra-Operative Images by Exploiting DataUncertainties”, U.S. Provisional Application No. 61/605,566 of Grbic, etal., filed Mar. 1, 2012, and “Ultrasound and Fluoroscopic images Fusionby Autonomous US Probe Detection”, U.S. Provisional Application No.61/605,573 of Mountney, et al., filed Mar. 1, 2012, the contents of allof which are herein incorporated by reference in their entireties.

TECHNICAL FIELD

This disclosure is directed to methods for real-time fusing of 2D and 3Dimages with 2D fluoroscopic images for interventional guidance.

DISCUSSION OF THE RELATED ART

Fluoroscopy guided cardiac interventions such as endovascular stenting,atrial ablation, closure of atrial/ventricular septal defects andtranscatheter valve repair or replacement are proliferating. Incomparison to conventional open-heart surgeries, these procedures tendto be less invasive, reduce procedural morbidity, mortality andinterventional cost while accelerating patient recovery. For inoperableor high-risk patients, minimal invasive cardiac intervention is the onlytreatment option. However, navigating a catheter inside a patient ischallenging, and without direct access or view to the affected anatomy,advanced imaging is required to secure a safe and effective execution ofthe procedure.

There are two established modalities currently used in operating roomsto provide real-time intra-operative images: X-ray fluoroscopy (Fluoro)and Transesophageal Echocardiography (TEE). X-ray fluoroscopy is used tovisualize the catheter; however, this imaging modality does not capturesoft tissue structure. Soft tissue is visualized using a second imagingmodality, e.g. Transesophageal Echocardiography (TEE), or contrast agentcombined with rapid pacing. Nevertheless, the splendid complementarynature of TEE and Fluoro is barely exploited in today's practice wherethe real-time acquisitions are not synchronized, and images arevisualized separately and in misaligned coordinate systems.

On the other hand, overlays of 3D anatomical structures based onpre-operative data can provide valuable information for interventionnavigation and guidance when displayed on 2D live fluoroscopy. Valuable3D information is already routinely acquired for diagnostic and planningpurposes by means of Computed Tomography, Magnetic Resonance Imaging(MRI) or Echocardiography. However, direct 3D to 2D image registrationis challenge to solve, especially within the intra-operative setup thatdoes not allow for user interaction or time consuming processing.

In a procedure such as Transcatheter Aortic Valve Implantation (TAVI),visualization of soft tissue is critical to ensure the correctplacement/alignment of the implant. TEE provides useful navigation data;however, it is normal to perform rotational angiography with rapidpacing or a contrast agent to obtain models of the soft tissuestructures. Overlaying rotational angiography on a fluoroscopic imageenables correct alignment of the device using fluoroscopy.

However, clinical guidelines limit the duration and frequency of rapidpacing and the volume of contrast agent that can be administered to apatient, due to negative effects on the heart and kidneys. Analternative approach is to visualize soft tissue information from TEE inthe fluoroscopic image. This will facilitate navigation of the implantdevice in fluoroscopy.

The fusion of fluoroscopic and ultrasound (US) images into a singlespace is challenging. Fluoroscopy is a projective imaging modality andUS is 2D or 3D. These modalities are not intuitively visualized in thesame space. In addition care must be taken to visualize meaningfulinformation and to not occlude important data.

The fusion of Fluoro and TEE can be accomplished using either hardwareor image-based methods. Hardware based approaches attach additionaldevices to the ultrasound probe, such as electromagnetic or mechanicaltrackers and align the device and Fluoro coordinates systems throughcalibration. These devices track the position and orientation of theprobe in a coordinate system defined by the tracking device. Through acalibration process, the transformation between the ultrasound image andthe tracked point on the probe is estimated. This transformation isrigid and does not change during the procedure. A second calibrationprocedure estimates the transformation between the tracking devicecoordinate system and the X-ray fluoroscopy device. Concatenating thesetransformations registers the ultrasound image into the X-rayfluoroscopy image. It is assumed that the ultrasound image is notrotated or zoomed.

The introduction of additional hardware into the already crowdedoperating theatre is not desirable, as it can require time consumingconfiguration and may be disruptive to the workflow. In additional,electromagnetic tracks can suffer from noise and interference leading toinaccuracies.

Image based methods attempt to use the appearance of the TEE probe inthe Fluoro image to estimate the pose of the probe in the fluoroscopiccoordinate system. Image based methods are attractive because they donot require the introduction of additional equipment into the theatrewhich may disrupt clinical workflow. Image based pose estimation is wellstudied and may be considered solved when the correspondence between 2Dimage points and a 3D model are known. Unfortunately, the appearance ofthe TEE probe in the Fluoro image makes establishing the correspondencechallenging. The probe's appearance lacks texture or clear featurepoints and can be homogenous under low dose or close to dense tissue.

C-arm CT is emerging as a novel imaging modality that can acquire 3DCT-like volumes directly in the operating room, in the same coordinatespace as the 2D live fluoroscopy images, which overcomes the need for2D/3D registration. Some methods work directly on the 3D C-arm CT imagesto extract patient specific models and overlays for procedure guidance,eliminating the need for pre- and intra-operative image fusioncompletely. However, performing high-quality, contrasted, and motioncompensated (using rapid-pacing) C-arm CT images is not feasible for allpatients. Instead, a much simpler protocol, which acquiresnon-contrasted, non-ECG-gated C-arm CT volumes, can be performed toserve as a bridge between 3D pre-operative images and 2D livefluoroscopy. Multi-modal 3D-3D registration algorithms can be utilizedto align the pre-operative image with the C-arm CT volume. FIG. 1depicts several fused images of an intra-operative 3D C-arm CT overlaidwith pre-operative model of the aortic valve extracted from CT. The CTis indicated by reference number 11 while an overlaid aligned nativerotational angiography is indicated by reference number 12. However,existing methods are computationally expensive, and without theappropriate guidance of a shape prior are unlikely to converge to localminima.

SUMMARY

Exemplary embodiments of the invention as described herein generallyinclude methods for fusing 3D pre-operative anatomical information withlive 2D intra-operative fluoroscopy via non-contrasted 3D C-arm CT.Embodiments employ robust learning-based methods to automaticallyextract patient-specific models of both target and anchor anatomies fromCT. Anchor anatomies have correspondences in the pre-operative andintra-operative images while target anatomies are not visible in theintra-operative image but are essential to the procedure. A sparsematching approach is employed to align the pre-operative anchoranatomies to the intra-operative setting. Data and model uncertaintiesare learned and exploited during the matching process. A methodaccording to an embodiment of the invention can cope with artifacts inthe intra-operative images, partially visible models and does notrequire contrast agent in the intra-operative image.

Further exemplary embodiments of the invention as described hereingenerally include methods for a robust and fast learning-based methodfor the automated detection and visualization of the TEE probe pose,with six degrees of freedom, from Fluoro images. Embodiments employ aprobabilistic model-based approach to estimate candidates for thein-plane probe position, orientation and scale parameters, and digitallyreconstructed radiography (DRR) in combination with a fast matchingbased on binary template representation for the estimation of out-planerotation parameters (pitch and roll). An approach according to anembodiment of the invention is an image only approach which requires noadditional hardware to be incorporated into the operating theatre, doesnot require manual initialization, is robust over the entire poseparameter space, and is independent of specific TEE probedesign/manufacturer. The 6 degree-of-freedom (DoF) pose of the probe canbe detected from 2D fluoroscopy enabling the ultrasound (US) fan to bevisualized in the same coordinate system as the fluoroscopy.

Further exemplary embodiments of the invention as described hereingenerally include methods for visualizing high contrast informationextracted from the US of anatomically significant structures,specifically the aortic root and leaflets, to facilitate implantguidance, and the pose of the US probe in the fluoroscopic image.Embodiments can meet real time requirements by detecting critical softtissue anatomy in 2D US images.

According to an aspect of the invention, there is provided a method forreal-time fusion of a 2D cardiac ultrasound image with a 2D cardiacfluoroscopic image, including detecting a surface contour of an aorticvalve in the 2D cardiac ultrasound (US) image relative to an US probe,detecting a pose of the US probe in the 2D cardiac fluoroscopic image,and using pose parameters of the US probe to transform the surfacecontour of the aortic valve from the 2D cardiac US image to the 2Dcardiac fluoroscopic image.

According to a further aspect of the invention, detecting the surfacecontour of the aortic valve includes modeling a global location of theaortic valve by a bounding box with a specified center and orientation,where the global location includes a center position, an orientation anda scale of the aortic valve, locating anatomical landmarks of the aorticvalve, including 2 landmarks on the aortic valve annulus and 2 landmarkson the aortic valve commissure plane, and modeling the aortic valveborders with a first contour and a second contour, the first and secondscontours being constrained by the aortic valve annulus landmarks and theaortic valve commissure plane landmarks.

According to a further aspect of the invention, the method includesdetecting the global position, anatomical landmarks, and first andsecond contours are using marginal space learning with a hierarchicalapproach, where detectors are successively trained using probabilisticboosting trees.

According to a further aspect of the invention, the method includesfinding an optimal imaging angle for the US probe by rotating the USprobe about its axis, and detecting an angulation of an US fan withrespect to the aortic root, and selecting a probe orientation thatmaximizes the angulation of the US fan with respect to the aortic rootas the optimal imaging angle.

According to a further aspect of the invention, the method includesinserting the US image into the fluoroscopic image.

According to a further aspect of the invention, detecting a pose of theUS probe in the 2D cardiac fluoroscopic image includes determining aposition (u,v), orientation (θy), and size (s) of an ultrasound (US)probe in a fluoroscopic image, determining a roll and pitch of the USprobe in the fluoroscopic image, where the position, orientation, size,roll and pitch comprise pose parameters of the probe, and using theprobe pose parameters to transform points in the 2D cardiac ultrasoundimage into the 2D cardiac fluoroscopic image, where the 2D cardiacultrasound image is visualized in the 2D cardiac fluoroscopic image.

According to a further aspect of the invention, determining theposition, orientation, and size of the US probe in the fluoroscopicimage comprises sequentially applying a classifier for each of theposition, orientation, and size, respectively, where each classifier istrained using a probabilistic boosting tree.

According to a further aspect of the invention, each of the classifiersis trained using Haar-like features.

According to a further aspect of the invention, determining the positionof the US probe comprises applying a steerable filter to the 2Dfluoroscopic image to identify regions of high contrast which are likelyto contain the US probe.

According to a further aspect of the invention, determining the size ofthe US probe comprises detecting two points where a tip of the probemeets a shaft of the probe, where the orientation and position of the USprobe are used to constrain a search area for the size detector.

According to a further aspect of the invention, determining the roll andpitch of the US probe in the fluoroscopic image comprises matching animage patch of the fluoroscopic image containing the US probe with eachof a plurality of image templates, where each image template isassociated with a particular combination of roll and pitch values, wherethe pitch and roll of a template that best matches the image patch areselected as the roll and pitch of the US probe.

According to another aspect of the invention, there is provided a methodof transforming target structure anatomies in a pre-operative image I₂into an intra-operative image I₁, including determining a transformationΦ aligns a target structure T₂ and an anchor structure A₂ in thepre-operative image I2 into a corresponding target structure T₁ andanchor structure A₁ in the intra-operative image I₁ by finding atransformation {circumflex over (Φ)} that maximizes a functionallog(P(Φ|I₁, A₂)) using an expectation-maximization approach, where thetarget structure T₁ is not visible in the intra-operative image.

According to a further aspect of the invention, the transformation Φ isa rigid transformation, where an initial transformation Φ⁰ isapproximated as a translation, where Φ⁰ represents a translation betweena barycenter a₂ of the anchor anatomy A₂ in the pre-operative image I₂and a detected barycenter a₁ of the anchor anatomy A₁ in theintra-operative image I₁.

According to a further aspect of the invention, the initialtransformation Φ⁰ is determined by a position detector trained by aprobabilistic boosting tree classifier and Haar features on thebarycenter a₁ of the anchor anatomy A₁ in the intra-operative image I₁.

According to a further aspect of the invention, the pericardium is usedas the anchor anatomy A₁ and A₂ and the aortic valve is used as thetarget anatomy T₁ and T₂.

According to a further aspect of the invention, finding a transformation{circumflex over (Φ)} that maximizes a functional log(P(Φ|I₁, A₂))includes generating K sample Φ_(i) ^(t) point sets (x₁, x₂, x₃, . . . ,x_(K)) from the pre-operative anchor anatomy A₂, where each point setcomprises N points and each sample is represented as an isotropic 6DGaussian distribution Φ_(i) ^(t)=N₆(μ_(i),Σ_(i)), Σ_(i)=σ_(i)I, where Iis an identity matrix and σ_(i) is a one dimensional variable calculatedas a kernel function from a probability map F(I) evaluated at the pointlocations y_(i,j), i=1, . . . , K, j=1, . . . , N, transforming thepoint sets Φ_(i) ^(t) into the intra-operative image I₁ locationsy*_(i)=Φ^(t)(x_(i)), i=1, . . . , K , according to an appearance of theintra-operative image I₁, assigning each point y*_(i,j), j=1, . . . , Nfrom the point set to a new location y_(i,j), j=1, . . . , N, based on alocal appearance of the intra-operative image I₁, approximating finalparameters of each sample Φ_(i) ^(t) by an isotropic Gaussiandistribution, where a mean μ is computed from a least squares solutionbetween the point set Φ_(i) ^(t) in the pre-operative I₂ and the updatedpoint set (y₁, y₂, y₃, . . . , y_(K)) in the intra-operative image I₁ byminimizing the mapping error function

${e_{i} = {\frac{1}{N}{\sum\limits_{j = 1}^{N}{{{\Phi_{i}^{t}( x_{i,j} )} - y_{i,j}}}}}},$

and determining an updated global transformation Φ^(t+1) from

$\Phi^{t + 1} = {\underset{\Phi}{\arg \; \max}( \Phi \middle| \Phi^{t} )}$

based on an estimated mixture model

$\oplus {= {\sum\limits_{i = 1}^{K}\Phi_{i}^{t}}}$

of the K transformation samples Φ_(i) ^(t), i=1, . . . , K.

According to a further aspect of the invention,

$\Phi^{t + 1} = {\underset{\Phi}{\arg \; \max}( \Phi \middle| \Phi^{t} )}$

is estimated using a mean shift algorithm.

According to a further aspect of the invention, the method includesderiving the probability map F(I₁) from the intra-operative image I₁ byevaluating a boosting classifier trained using Haar features and surfaceannotations of the anchor anatomy A₁ in the intra-operative image I₁,where each vertex of a model of the intra-operative image I₁ is assignedas a positive sample and random points within a threshold distance areused as negative samples, and those vertices for which a featureresponse is low are rejected as positive examples.

According to a further aspect of the invention, minimizing the mappingerror further comprises estimating a prior probability for each vertexof a model of pre-operative image I₂ by assigning each vertex of a modelof the pre-operative image I₂ as a positive sample and using randompoints within a threshold distance as negative samples, rejecting thosevertices for which a feature response is low as positive examples,estimating a ground-truth mapping Φ_(T) based on hinges and commissuresof the aortic valve, and transforming each intra-operative model of thepre-operative anchor anatomy T₂ into the pre-operative image I₁ usingT₁*=Φ_(T)T₂ and the variance of a point-wise distance ∥T₁*−T₁∥.

According to another aspect of the invention, there is provided anon-transitory program storage device readable by a computer, tangiblyembodying a program of instructions executed by the computer to performthe method steps for transforming target structure anatomies in apre-operative image I₂ into an intra-operative image I₁.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts several fused images of an intra-operative 3D C-arm CToverlaid with pre-operative model of the aortic valve extracted from CT,according to an embodiment of the invention.

FIG. 2 is flowchart of a method for real time image fusion according toan embodiment of the invention.

FIGS. 3( a)-(b) depict a 2D ultrasound image of the aortic valve and abounding box showing the global position of the aortic valve accordingto an embodiment of the invention.

FIGS. 4( a)-(b) depict a landmark model of the aortic valve, and acomplete model of the aortic valve including the contours constrained bythe bounding box and the landmarks according to an embodiment of theinvention.

FIG. 5 is a schematic visualization of an optimization process accordingto an embodiment of the invention.

FIG. 6 depicts an example of a picture-in-picture visualization,according to an embodiment of the invention,

FIG. 7 shows a framework for determining in-plane and out-planeparameters, according to an embodiment of the invention.

FIG. 8 illustrates a formulation of fusing a pre-operative CT image intoan intra-operative 3D C-arm CT image, according to an embodiment of theinvention.

FIGS. 9( a)-(b) illustrates a model estimation derived from apre-operative CT using discriminative machine learning techniques forthe aortic valve and pericardium surface model, according to anembodiment of the invention.

FIG. 10 shows the output of the boosting classifier response on theintra-operative 3D C-arm CT data trained to delineate certain boundaryregions of the anchor anatomy, according to an embodiment of theinvention.

FIG. 11 illustrates one iteration of an EM approach according to anembodiment of the invention.

FIG. 12 illustrates prior weights indicating the significance of eachvertex for an accurate mapping with respect to the aortic valveaccording to an embodiment of the invention.

FIG. 13 is a table showing quantitative validation of the in-planeposition and orientation parameters for three datasets, according to anembodiment of the invention.

FIGS. 14( a)-(d) are fluoroscopic images illustrating probe detectionand the estimation of in-plane parameters from in vivo images, accordingto an embodiment of the invention.

FIG. 15 plots the (θr, θp) error in mm over the search space in degrees,according to an embodiment of the invention.

FIG. 16 is a table of the quantitative validation results for TEE probedetection, according to an embodiment of the invention.

FIG. 17 depicts detection examples of the probe pose in in vivo images,according to an embodiment of the invention.

FIGS. 18( a)-(c) illustrate an anatomical mitral valve model detected in3D TEE and visualized in Fluoro, according to an embodiment of theinvention.

FIG. 19 is a table of the mean, median and standard deviations ofvarious transformations, according to an embodiment of the invention.

FIG. 20 shows several examples of fused volumes with a mapped aorticvalve model detected in a pre-operative CT image mapped into anon-contrasted intra-operative 3D C-arm CT image using a sparse matchingmethod with prior sampling according to an embodiment of the invention.

FIG. 21 is a block diagram of an exemplary computer system for fusingimages for interventional guidance, according to an embodiment of theinvention.

FIG. 22 is a flowchart of a method of matching an image patch with theimage templates in the template library, according to an embodiment ofthe invention.

FIG. 23 is a flowchart of a method of deriving the probability map F(I₁)from the intra-operative image I₁, according to an embodiment of theinvention.

FIG. 24 is a flowchart of a method of estimating a prior probability foreach mesh point of the pre-operative image models, according to anembodiment of the invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Exemplary embodiments of the invention as described herein generallyinclude systems and methods for fusing images for interventionalguidance. Accordingly, while the invention is susceptible to variousmodifications and alternative forms, specific embodiments thereof areshown by way of example in the drawings and will herein be described indetail. It should be understood, however, that there is no intent tolimit the invention to the particular forms disclosed, but on thecontrary, the invention is to cover all modifications, equivalents, andalternatives falling within the spirit and scope of the invention.

As used herein, the term “image” refers to multi-dimensional datacomposed of discrete image elements (e.g., pixels for 2-dimensionalimages and voxels for 3-dimensional images). The image may be, forexample, a medical image of a subject collected by computer tomography,magnetic resonance imaging, ultrasound, or any other medical imagingsystem known to one of skill in the art. The image may also be providedfrom non-medical contexts, such as, for example, remote sensing systems,electron microscopy, etc. Although an image can be thought of as afunction from R³ to R or R⁷, the methods of the inventions are notlimited to such images, and can be applied to images of any dimension,e.g., a 2-dimensional picture or a 3-dimensional volume. For a 2- or3-dimensional image, the domain of the image is typically a 2- or3-dimensional rectangular array, wherein each pixel or voxel can beaddressed with reference to a set of 2 or 3 mutually orthogonal axes.The terms “digital” and “digitized” as used herein will refer to imagesor volumes, as appropriate, in a digital or digitized format acquiredvia a digital acquisition system or via conversion from an analog image.

Methods

Methods according to an embodiment of the invention can extract criticalsoft tissue information from 2D TEE in real time. A flow chart of amethod for real time image fusion according to an embodiment of theinvention is shown in FIG. 2. In a first step 20 real time synchronizedUS and fluoroscopic images are acquired. The 2D US image is processed todetect the aortic anatomy. The anatomy includes but is not limited tothe aortic root and the leaflets. To visualize the detected anatomy inthe fluoroscopic image, at step 21 the anatomy is estimated relative tothe probe using the angulation of the US fan, and at step 22 the 6 DoFpose of the probe is estimated from the fluoroscopic image. Given thisinformation and the intrinsic calibration parameters of the fluoroscopicdevice, the detected anatomy can be projected into the fluoroscopicimage plane at step 23 and visualized. FIG. 3( a) depicts a 2D US imageof the aortic valve.

Embodiments of the invention can detect the surface contour of theaortic valve in 2D (TEE) ultrasound using a multi-level hierarchicalapproach. On a coarsest layer, the location, orientation and scale aremodeled as a bounding box θ, as shown in FIG. 3( b), where θ={c₁, c₂,α), where c₁, c₂ represent the x- and y-coordinates of the bounding boxcenter and a the rotation. The size or scale of the bounding box islearned from the modeling.

A second modeling layer according to an embodiment of the inventionincludes four landmarks (m_(A1), m_(A2), m_(C1), m_(C2)) where m_(A1)and m_(A2) are located on the aortic valve annulus and m_(C1) and m_(C2)on the commissure plane. FIG. 4( a) depicts a landmark model of theaortic valve, according to an embodiment of the invention. A thirdmodeling layer according to an embodiment of the invention includes twocontours R1 and R2 which are constrained by the bounding box, annulusand commissures landmarks. FIG. 4( b) depicts a complete model of theaortic valve including the contours R1 and R2, according to anembodiment of the invention

According to embodiments of the invention, patient-specific parametersof the aortic valve model can be estimated from the 2D or 2×2D (X-Plane)ultrasound images using robust learning-based algorithms that usehierarchical approaches within a Marginal Space Learning (MSL) approach.Detectors are successively trained using the Probabilistic Boosting Tree(PBT) with Haar and Steerable features, and are subsequently applied toestimate the global location θ followed by anatomical landmarks (m_(A1),m_(A2), m_(C1), m_(C2)) and surface structures R1 and R2.

A model according to an embodiment of the invention is estimated in theultrasound image space and can therefore be transformed into thefluoroscopic image space using the approach described above.

An approach to US probe pose estimation according to an embodiment ofthe invention first detects the probe in the fluoroscopic image withthree degrees of freedom, two translational degrees and one rotationdegree, in the image plane. According to an embodiment of the invention,the probe can be detected using Marginal Space Learning andProbabilistic Boosting trees. A classifier according to an embodiment ofthe invention can be trained on manually labeled data, and can extractfeatures which distinguish positively labeled data from negativelylabeled data. Embodiments use non-maximal suppression to reduce thenumber of candidates, and boot strapping to initialize a detection andtracking process according to an embodiment of the invention.

A pose estimation according to an embodiment of the invention has 6 DoF.According to an embodiment of the invention, the remaining 3 degrees offreedom can be estimated using a second classifier. The secondclassifier can be trained to estimate the Z translation (depth), pitchand roll of the probe. The classifier is trained on syntheticallygenerated training data where the ground truth position of the probe isknown. According to an embodiment of the invention, a filter such as anExtended Kalman or a Particle filter can be used to exploit temporalinformation between frames, which reduces the search space, enabling thepose of the probe to be predicted.

A new clinical workflow according to an embodiment of the invention candetermine an optimal US orientation for visualization of the aorticroot. Detection of the aortic root in 2D US is beneficial for real timecomputation, however, the detected segments are only a partialrepresentation of the root. To visualize the whole root structure, anoperator can move the US device and determine an optimal imaging planefor visualization of the aortic structures in fluoroscopy. An optimalimaging angle is one which visualizes a widest point of the aortic rootand thus facilitates implantation of a prosthetic device.

FIG. 5 is a schematic visualization of an optimization process accordingto an embodiment of the invention. This figure represents a simple casein which the US probe is rotated around its axis, changing theangulations of the US fan with respect to the aortic root. Three stepsof a continuous motion are shown in FIG. 5. The user starts with aninitial visualization in fluoroscopy 51 of the aortic root representedby two lines. At this point it would not be clear to the operator ifthis is an optimal visualization, i.e., the widest point of the aorticroot. By rotating the probe around the axis, as shown on the left sideof FIG. 5, the operator can see the lines move further apart 52 as anoptimal visualization plane is approached. These lines will then moveback together 53 after passing the optimum visualization plane. Throughguided navigation and exploration the operator can determine the optimalimaging plane.

It should be noted that the aortic anatomy may not always be visualizedas straight or parallel lines. The visualization is dependent on the 6DoF orientation of the US probe and the shape of the anatomy. This doesnot affect the effectiveness of the navigation or the usefulness of thevisualization to assist in determining an optimal orientation of the USprobe, as it is still possible to visualize the widest part of theaortic root.

According to an embodiment of the invention, a picture-in-picturevisualization can enable a physician to verify the correctness of thedetected anatomy, and to verify that models visualized in the fluorocorrespond to that in the TEE. FIG. 6 depicts an example of apicture-in-picture visualization, according to an embodiment of theinvention, with US picture 61 embedded in the upper right corner of theimage. FIG. 6 also depicts the aortic root 62 and the aortic valve 63.

A method of fusing 2D TEE images with 2D fluoroscopic images can reducethe need for rapid pacing, reduce the use of a contrast agent, decreaseprocedure times, guide an ultrasound operator to find an optimal imagingplane, and provide a clear visualization of anatomy, by overlaying a TEEimage on a fluoroscopic image.

According to another embodiment of the invention, information from a TEEvolume can be visualized in a fluoroscopic image by aligning the TEE andC-arm fluoroscopic coordinate systems. A point Q^(TEE) in an ultrasoundvolume can be visualized in a fluoroscopic image at coordinate (u,v)=Q^(Fluoro) using a following transformation, according to anembodiment of the invention:

Q _(Fluoro) =P _(Projection) R _(xz) T _(d) R _(γ) R _(α)(R _(TEE) ^(W)Q ^(TEE) +T _(TEE) ^(W))  (1)

where P_(Projection) is a projection matrix, R_(xz) and T_(d) are thetransformations from a detector to a world coordinate system, R_(γ) andR_(α) are the angulations of the C-arm, and R_(TEE) ^(W) and T_(TEE)^(W) are the rotation and position of the TEE probe in a worldcoordinate system such that R_(TEE) ^(W)=R_(α) ⁻¹R_(γ) ⁻¹R_(xz)⁻¹R_(TEE) ^(Fluro) and T_(TEE) ^(w)=R_(α) ⁻¹R_(γ) ⁻¹R_(xz) ⁻¹R_(TEE)^(Fluro). The TEE volume and fluoroscopic image can be aligned ifposition T_(TEE) ^(Fluoro)=(x,y,z) and orientation R_(TEE)^(Fluoro)=(θr,θp,θy) of the TEE probe in the Fluoroscopic detector TEEcoordinates.

An approach according to an embodiment of the invention separates thepose parameters into in-plane (x, y, z) and (θy) parameters andout-plane (θr, θp) parameters. By marginalizing the estimation,embodiments can efficiently estimate in-plane parameters directly fromthe Fluoro images, while being invariant against the out-planeparameters that are more challenging to determine. A framework accordingto an embodiment of the invention for determining in-plane and out-planeparameters is illustrated in FIG. 7.

According to an embodiment of the invention, the in-plane parameters canbe computed from the position (u, v), size (s) and orientation (θy),given a projection transformation P of the calibration information ofthe fluoroscopic device and the physical dimensions of the TEE probe.Embodiments of the invention can detect the in-plane parameters (u, v),(s), (θy) from a Fluoro image using discriminative learning methodsdescribed below.

According to an embodiment of the invention, to estimate the in-planeparameters, discriminative learning methods can be used to train aclassifier that detects the position (u, v), the orientation (θy), andthe size (s) of the TEE probe in the Fluoro image. Three classifiers canbe trained using manually annotated Fluoro data. According to anembodiment of the invention, the classifiers are trained andsequentially applied so that first, candidates 71 a are detected for (u,v) at step 71, then the orientation (θy) 72 a is detected for eachcandidate at step 72, and finally the size 73a of the probe is detected(s) at step 73.

Each detector is a Probabilistic Boosting Tree (PBT), a binaryclassifier. According to an embodiment of the invention, each detectoris trained using Haar-like and steerable features. A position (u, v)detector according to an embodiment of the invention is trained onmanual annotations and negative examples randomly extracted from thefluoroscopic image. An exemplary, non-limiting fluoroscopic image isresized to 128×128 and a 35×35 window is centered at the annotation.100,000 Haar features are used to train the PBT. The appearance of theprobe varies greatly and to avoid over fitting, embodiments create aclassifier which is less discriminative but more likely to detect thetip of the probe. During detection a steerable filter is applied to theimage to identify regions of high contrast which are likely to containthe TEE probe. This reduces the number of image patches to be classifiedby the probe and improves speed.

An orientation (θy) detector according to an embodiment of the inventionis trained on manually annotated data and the false positives from theposition detector. Additional negative training data is created centeredon the annotation but with incorrect rotation parameters. A PBTaccording to an embodiment of the invention can be trained with fivefeatures, including the relative intensity and the difference betweentwo steerable filters applied to the image with different parameters. Anorientation detector according to an embodiment of the invention istrained at intervals of six degrees with a 360 degree coverage. Anorientation detector according to an embodiment of the invention is morediscriminative than the position detector and therefore can removeoutliers as well as estimating the orientation.

A size (s) detector according to an embodiment of the invention istrained to detect two points where the tip of the probe meets the shaft.This part of the probe is circular and appears the same size invariantof the pose. A PBT according to an embodiment of the invention can betrained using Haar features. During detection the orientation andposition of the probe are used to constrain the search area for the sizedetector.

The out-plane parameters are more challenging to estimate. Theappearance of the probe under roll and pitch (θr, θp) variessignificantly in the fluoroscopic image and cannot generally beaccounted for in the image space using the same techniques as used forthe in plane parameters, making it challenging to train a compactclassifier. Embodiments of the invention take a different approach bycreating a template library of fluoroscopic images of the probe underdifferent out-of-plane orientations (θr, θp). Referring again to FIG. 7,at step 74, the (θr, θp) parameters 74 a, 74 b are estimated by matchingan image patch, normalized for the in-plane parameters, with thetemplate library. Each template has an associated (θr, θp) and bymatching the fluoroscopic image to the template at step 75 one canestimate the out-of-plane parameters as below. The TEE probe can bevisualized at step 76.

A template library according to an embodiment of the invention shouldcontain a wide variety of orientations. It is not feasible to build thislibrary from in vivo data as it is challenging to manually annotate (θr,θp) and the data may not be comprehensive. Embodiments build a libraryusing Digitally Reconstructed Radiography (DRR). DRR's can simulateX-ray fluoroscopy by tracing light rays through a 3D volume. For thispurpose, a 512×512×433 rotational angiography of the TEE probe isacquired with a 0.2225 mm resolution. The orientation and position ofthe probe is manually annotated and (θr, θp) orientations are applied tothe volume. Generating DRR images is computationally expensive andmoving this stage offline saves computation online.

Searching a template library according to an embodiment of the inventioncan be computationally expensive. The size of the library can be limitedto reduce the search space. The probe is not free to move in alldirections due to physical constraints of the tissue. In addition, theX-ray image is an integral image and is therefore reflective. These twofacts can be exploited by embodiments to reduce the size of the templatelibrary. According to an embodiment of the invention, a library wasbuilt with pitch poses from −45 to 45 degrees and roll poses from −90 to90 degrees with two degree intervals. The library includes 4050 imagepatches. These values are exemplary and non-limiting, and templatelibraries can be built over different angular ranges with differentangular intervals in other embodiments of the invention.

This subsample library is still large and expensive to store and search.To make searching computationally tractable, embodiments use a binarytemplate representation. Binary templates are an efficient way ofstoring information about an image patch which can be useful formatching. In addition because the information is stored in binary,matching can be quickly performed using bitwise operations.

A flowchart of a method according to an embodiment of the invention ofmatching an image patch with the image templates in the template libraryis presented in FIG. 22. Referring now to the figure, the image patchcan be divided into sub-regions at step 221 and features can beextracted from each region at step 222. The dominant gradientorientation in each subregion is taken to be a feature, which works wellon homogenous regions and objects which lack texture, as is the case fora TEE probe in the fluoroscopic image. The orientations can bediscretized into N orientation bins at step 223. Each sub-region can berepresented as an N-bit byte which corresponds to the N orientationbins. An exemplary, non-limiting value for N is 8. At step 224, the bitis set to 1 if the orientation exists in the sub-region and 0 if it doesnot. The binary template for the image patch is comprised of a set ofbytes corresponding to the sub-regions. The resulting template is acompact and discriminative representation of the image patch.

According to an embodiment of the invention, templates are matched atstep 225 by comparing each sub-region and counting how many times afeature exists in the template and the input image. There is nomeasurement of the similarity of the features, only that a featureexists in a sub-region. The similarity measure is

$\begin{matrix}{{{ɛ( {I^{Fluoro},O,c} )} = {\sum\limits_{r}{\delta ( {{F( {{I^{Fluoro}( {u,v} )} + r} )} = {F( {O,r} )}} )}}},} & (2)\end{matrix}$

where δ(P) is a binary function which returns true if two featuresmatch, F(I^(Fluoro)(u,v)+r) is the input template centered on candidate(u,v) in image I^(Fluoro) and F(O, r) is a template from the templatelibrary. This function can be evaluated very quickly using a bitwise ANDoperation followed by a bit count. The final matching score is the bitcount and the (θr, θp) associated with the highest matching template isused to estimate the out-of-plane parameters.

According to another embodiment of the invention, a transformation Φbetween the target structure anatomies T₁ and A₁ in an intra-operativeimage I₁, and source structure anatomies T₂ and A₂ in a pre-operativeimage I₂ can be estimated:

(T ₁)=Φ(T ₂ ,A ₂).  (3)

FIG. 8 illustrates a fusion formulation according to an embodiment ofthe invention, showing the target T₁ and T₂ and anchor A₁ and A₂anatomies. The transformation matrix Φ maps the pre-operative CT imageI₁ to the intra-operative 3D C-arm CT image I₂.

Following the chronology of a typical clinical workflow, pre-operativestructures A₂ and T₂ are treated as an input for the remainder of thisdisclosure. According to an embodiment of the invention, the pericardiumis used as the anchor anatomy A₁ and A₂ and the aortic valve is used asthe target anatomy T₁ and T₂. All models are estimated using robust,discriminative learning based methods, and final model estimations frompre-operative CT images I₂ are shown in FIG. 8. The precision of thefinal surface model for the pericardium is 1.91 mm±0.71 and for theaortic valve is 1.21 mm±0.21. FIGS. 9( a)-(b) illustrates a modelestimation derived from a pre-operative CT I₂ using discriminativemachine learning techniques for the aortic valve and pericardium surfacemodel. FIG. 9( a) shows the aortic valve root, the leaflet tips, thehinges, the commissure points, and the ostias, while FIG. 9( b) showsthe pericardium surface mesh model.

A method according to an embodiment of the invention can find an optimaltransformation Φ that aligns the pre-operative structures T₂ and A₂ tothe intra-operative image I₁:

$\begin{matrix}{\hat{\Phi} - {\underset{\Phi}{\arg \; \max}\; {{\log ( {P( { \Phi \middle| I_{1} ,A_{2}} )} )}.}}} & (4)\end{matrix}$

The target structure T₁ is not visible in the intra-operative image, andtherefore the transformation Φ is determined only through the anchorstructures. Embodiments of the invention model Φ as a rigidtransformation with six degrees of freedom.

The initial transformation Φ⁰ is approximated as a translation. Aposition detector can be trained using a probabilistic boosting treeclassifier and Haar features on the barycenter a₁ of the anchor anatomyA₁ in the intra-operative image I₁. Thus Φ⁰ represents the translationbetween the barycenter a₂ of the anchor anatomy A₂ in the pre-operativeimage I₂ and the detected barycenter a₁ in the intra-operative image I₁.

According to an embodiment of the invention, an expectation-maximization(EM) framework is used to determine the final parameters. FIG. 11illustrates one iteration of an EM approach according to an embodimentof the invention to estimate the parameters of the transformationΦ^(t+1).

Referring now to FIG. 11, in an expectation stage, given a currentestimate of the global transformation Φ^(t), K samples Φ_(i) ^(t) pointsets (x₁, x₂, x₃, . . . , x_(x)) are generated at step 111 from thepre-operative anchor anatomy A₂, where each point set comprises N pointsand each sample is represented as an isotropic 6D Gaussian distributionwith μ_(i) representing the rigid transformation parameters and Σ_(i)the uncertainty of the sample:

Φ_(i) ^(t) =N ₆(μ_(i),Σ_(i)), Σ_(i)=σ_(i) I,  (5)

where I is the identity matrix and σ_(i) is a one dimensional variablecalculated as a kernel function from the probability map F(I) evaluatedat the point locations y_(i,j), i=1, . . . , K, j=1, . . . , N.

Given the current estimate of the transformation Φ_(i) ^(t) the pointsets are transformed at step 112 into the intra-operative image I₁.

y _(i)*=Φ^(t)(x _(i)), i=1, . . . , K.  (6)

The mapped point sets are updated according to the image appearance ofthe intra-operative image I₁. Each point y_(i,j)*, j=1, . . . , N fromthe point set x_(i) is assigned a new location y_(i,j), j=1, . . . , Nbased on the local image appearance.

According to an embodiment of the invention, to secure a robust updateschema a probability map F(I₁) is used, which is derived from theintra-operative image I₁ by evaluating a boosting classifier trainedusing Haar features. The classifier can be trained using surfaceannotations of the anchor anatomy A₁ in the intra-operative image. Aflowchart of a method according to an embodiment of the invention ofderiving the probability map F(I₁) from the intra-operative image I₁ ispresented in FIG. 23. Referring now to the figure, each vertex can beassigned as a positive sample at step 231 and random points within athreshold distance can be used as negative samples at step 232. At step233, those vertices for which the feature response of the Hessian is loware rejected as positive examples. The classifier is trained on theremaining samples at step 234. The output of the boosting classifierresponse on the intra-operative 3D C-arm CT data F(I₁) trained todelineate certain boundary regions of the anchor anatomy A1(pericardium) is shown in FIG. 10. Uncertain regions 101 such as theboundary between the pericardium and the liver have low response whilethe transition 102 from the pericardium and the lung have highconfidence.

Referring again to FIG. 11, the final parameters of each sample Φ_(i)^(t) are approximated at step 113 by an isotropic Gaussian distribution.The mean μ is computed from a least squares solution between the pointset in the pre-operative data (x₁, x₂, x₃, . . . , x_(K)) and theupdated point set (y₁, y₂, y₃, . . . , y_(K)) in the intra-operativeimage I₁ by minimizing the mapping error function e_(i)

$\begin{matrix}{e_{i} = {\frac{1}{N}{\sum\limits_{j = 1}^{N}{{{{\Phi_{i}^{t}( x_{i,j} )} - y_{i,j}}}.}}}} & (7)\end{matrix}$

In a maximization stage, the values of the global transformation Φ^(t)are updated at step 114 based on the estimated mixture model

$\oplus {= {\sum\limits_{i = 1}^{K}\Phi_{i}^{t}}}$

of the K transformation samples Φ_(i) ^(t), i=1, . . . , K:

$\begin{matrix}{\Phi^{t + 1} = {\underset{\Phi}{\arg \; \max}( \Phi \middle| \Phi^{t} )}} & (8)\end{matrix}$

As there is no analytic solution, embodiments employ a mean shiftalgorithm to approximate the solution.

To minimize a mapping error with regard to the target anatomy T₁ and T₂,embodiments estimate a prior probability for each mesh point of thepre-operative image models. According to an embodiment of the invention,this information can be incorporated into the expectation phase whererandom points can be sampled on the pre-operative anchor model A₂. Aflowchart of a method according to an embodiment of the invention ofestimating a prior probability for each mesh point of the pre-operativeimage models is presented in FIG. 24. Referring now to the figure, eachvertex can be assigned as a positive sample at step 241 and randompoints within a threshold distance can be used as negative samples atstep 242. At step 243, those vertices for which the feature response ofthe Hessian is low are rejected as positive examples. Based on the 3aortic valve hinges and the 3 aortic valve commissures, depicted in FIG.9( a), a ground-truth mapping Φ_(T) is estimated at step 245. At step246, every intra-operative model of the pre-operative anchor anatomy T₂is transformed to the pre-operative image I₁ using T₁*Φ_(T)T₂ and thevariance of the point-wise distance ∥T₁*−T₁∥.

FIG. 12 illustrates prior weights indicating the significance of eachvertex for an accurate mapping with respect to the target anatomy T, theaortic valve. Reference number 121 indicates high probability regionswhile reference number 122 indicates low probability locations. Most ofthe significant area is located around the left atrium, while the leftventricle shows low confidence for the location of the aortic valve.FIG. 12 confirms that certain regions on the anchor anatomy may betterapproximate the desired transformation Φ between the target anatomiesthan others. Points on the left atrium may align the pre-operative andintra-operative images with respect to the target anatomy T₁ and T₂, theaortic valve.

EXPERIMENTS

A method according to an embodiment of the invention for probe posedetection was validated on synthetic, phantom and in vivo datasets.Throughout the experiments a GE Linear TTE Transducer was used. Thesynthetic dataset includes 4050 simulated fluoroscopy images generatedby means of DRR from a 3D C-arm rotational angiography volume of the TEEprobe, which cover the entire search space of out-plane parameters. Thevolume size was 512×512×4330 with 0.2225 mm per slice. The ground-truthwas generated by annotating the 3D probe position in the rotationalangiography volume and projecting it into the simulated fluoroscopyimages. The phantom dataset includes a rotational angiography volume ofthe TEE probe inserted into a silicon phantom, and a total of 51fluoroscopic images captured by rotating the C-arm and keeping the TEEprobe static. The position of the C-arm is known from the roboticcontrol, which enabled the ground-truth to be computed for eachfluoroscopic image from a 3D probe annotation, similar to the syntheticdata. The in vivo dataset was acquired during several porcine studiesand includes 50 fluoroscopic sequences comprising of about 7,000 frames,which cover an extensive range of probe angulations. The pose parameterswere manually annotated in all sequences and corresponding frames, andassumed as ground-truth for training and testing.

In a first experiment, the quantitative and qualitative performanceevaluation of the in-plane parameter (u, v, θy) detection was performedon all three datasets. The detector was trained on 75% of the in vivodataset (36 sequences of 5,363 frames) and tested on the entiresynthetic, phantom and remaining 25% of the in vivo dataset. The resultsare summarized in Table 1, shown in FIG. 13. In the table, the numbersin parentheses are standard deviations.

For the in vivo data the average in-plane position (u, v) error was 2.2and 3.7 mm, respectively, and the in-plane orientation error was 6.69degrees. Errors in the position estimation are caused by falsedetections along the shaft of the probe. False position detectionscontribute to errors in the orientation estimation. The true positiverate is 0.88 and the false positive rate is 0.22. The detection andaccuracy is affected by dose level, proximity to dense tissue andbackground clutter. For a detection framework according to an embodimentof the invention, the probe should be clearly distinguishable from itsbackground. FIGS. 14( a)-(d) illustrate detection examples and nature ofin vivo images with cluttered background and low textured probe, asindicated by the box and arrow 140 in each image.

The results for the phantom and synthetic data are provided in Table 1where detection was performed at a fixed scale. The Fluoro data from thephantom experiment appears different from the in vivo data used to trainthe detectors making it challenging. The true positive rate was 0.95 andfalse positive rate 0.05. False detections were caused by the density ofthe silicon phantom, which obscures the probe in three images. The truepositive and false positive rates for synthetic data were 0.99 and 0.01respectively. The visual appearance of the synthetic DRR is differentfrom the training data, however the probe is distinguishable causinghigh true positive rate.

The out-of-plane (θr, θp) detectors are analyzed on the synthetic datato evaluate the accuracy of the binary template matching. FIG. 15 plotsthe (θr, θp) error in mm over the search space in degrees andillustrates stable detection with a single outlier.

Finally a framework according to an embodiment of the invention wasevaluated with respect to all parameters. Quantitative validation wasperformed on synthetic and phantom data, as ground truth data for invivo data was not available. The results are summarized in Table 2,shown in FIG. 16. In the table, the numbers in parentheses are standarddeviations. The largest error is in the Z axis, which corresponds to theoptical axis of the Fluoro device. It is expected that this would be thelargest error because estimating distance along the optical axis ischallenging from a monocular Fluoro image. Fortunately, the goal of theframework is to visualize anatomy in the Fluoro image, therefore errorsin Z has little effect on the final visualization. Qualitativeevaluation is performed on in vivo Fluoro images, depicted in FIG. 17,which are Fluoro images showing the detected pose of the probe,indicated by the arrows 170.

The computational performance was evaluated for an Intel 2.13 GHz singlecore with 3.4 GB of RAM. The average detection time is 0.53 seconds. Thecomputational cost can be reduced by incorporating temporal informationto reduce the search space.

To illustrate the clinical relevance of a method according to anembodiment of the invention, an anatomical model of the mitral valve isdetected in a 3D TEE and visualized in Fluoro. FIG. 18( a) is a Fluoroimage of the catheter, FIG. 18( b) depicts the mitral detected in 3DTEE, and FIG. 18( c) shows the valve model visualized in Fluoro. Themodalities are not synchronized and are manually fused. The catheter 180can be seen in both modalities.

A further experiment was performed to validate a mapping Φ according toan embodiment of the invention from pre-operative CT to anintra-operative 3D C-arm CT used 37 patient pairs (74 volumes).According to an embodiment of the invention, contrasted intra-operative3D C-arm CT were used as the aortic valve can be manually annotated andused for quantitative comparisons. All ground-truth annotations wereobtained by expert users manually placing anatomical landmarks and thefull surface models of the target and anchor anatomies in the pre- andintra-operative images. The estimation errors can be assessed from TableIII, shown in FIG. 19, which displays the system precision for theestimation of target anatomy T₁=Φ(T₂). The error is evaluated as thedeviation of the transformed target anatomy Φ(T₂) and the ground-truthannotation T_(1,GT). A sparse matching method according to an embodimentof the invention using a prior sampling scheme has the best performance.It is more accurate than standard rigid registration algorithms using amutual information metric, and the transformation Φ extracted from theannotated anchor anatomies A₁ and A₂. The reason may be that a C-arm CTcontains many uncertain regions of the anchor anatomy without clearcontrast at the anatomy border. Thus, user annotations are notconsistent between the two modalities and produce a larger mapping errorthan our fully automated method.

FIG. 20 shows several examples of fused volumes with a mapped aorticvalve model 201 detected in pre-operative CT I₂ and mapped into thenon-contrasted intra-operative 3D C-arm CT I₁ using a sparse matchingmethod with prior sampling according to an embodiment of the invention.For clarity, the model outline 201 is indicated in only one of theimages of FIG. 20. The first row shows different volume cuts with theestimated target T₂ and anchor A₂ anatomies. The middle and bottom rowsshow the aligned anchor A₁ and target T₁ anatomies. Left image shows anexample of 1.86 mm and right 4.03 mm error of the mapped target anatomywhen compared to the ground truth annotation.

System Implementations

It is to be understood that embodiments of the present invention can beimplemented in various forms of hardware, software, firmware, specialpurpose processes, or a combination thereof. In one embodiment, thepresent invention can be implemented in software as an applicationprogram tangible embodied on a computer readable program storage device.The application program can be uploaded to, and executed by, a machinecomprising any suitable architecture.

FIG. 21 is a block diagram of an exemplary computer system forimplementing a system for fusing images for interventional guidance,according to an embodiment of the invention. Referring now to FIG. 21, acomputer system 211 for implementing the present invention can comprise,inter alia, a central processing unit (CPU) 212, a memory 213 and aninput/output (I/O) interface 214. The computer system 211 is generallycoupled through the I/O interface 214 to a display 215 and various inputdevices 216 such as a mouse and a keyboard. The support circuits caninclude circuits such as cache, power supplies, clock circuits, and acommunication bus. The memory 213 can include random access memory(RAM), read only memory (ROM), disk drive, tape drive, etc., or acombinations thereof. The present invention can be implemented as aroutine 217 that is stored in memory 213 and executed by the CPU 212 toprocess the signal from the signal source 218. As such, the computersystem 211 is a general purpose computer system that becomes a specificpurpose computer system when executing the routine 217 of the presentinvention.

The computer system 211 also includes an operating system and microinstruction code. The various processes and functions described hereincan either be part of the micro instruction code or part of theapplication program (or combination thereof) which is executed via theoperating system. In addition, various other peripheral devices can beconnected to the computer platform such as an additional data storagedevice and a printing device.

It is to be further understood that, because some of the constituentsystem components and method steps depicted in the accompanying figurescan be implemented in software, the actual connections between thesystems components (or the process steps) may differ depending upon themanner in which the present invention is programmed. Given the teachingsof the present invention provided herein, one of ordinary skill in therelated art will be able to contemplate these and similarimplementations or configurations of the present invention.

While the present invention has been described in detail with referenceto exemplary embodiments, those skilled in the art will appreciate thatvarious modifications and substitutions can be made thereto withoutdeparting from the spirit and scope of the invention as set forth in theappended claims.

What is claimed is:
 1. A method for real-time fusion of a 2D cardiacultrasound image with a 2D cardiac fluoroscopic image, comprising thesteps of: detecting a surface contour of an aortic valve in the 2Dcardiac ultrasound (US) image relative to an US probe; detecting a poseof the US probe in the 2D cardiac fluoroscopic image; and using poseparameters of the US probe to transform the surface contour of theaortic valve from the 2D cardiac US image to the 2D cardiac fluoroscopicimage.
 2. The method of claim 1, wherein detecting the surface contourof the aortic valve comprises: modeling a global location of the aorticvalve by a bounding box with a specified center and orientation, whereinsaid global location includes a center position, an orientation and ascale of the aortic valve; locating anatomical landmarks of the aorticvalve, including 2 landmarks on the aortic valve annulus and 2 landmarkson the aortic valve commissure plane; and modeling the aortic valveborders with a first contour and a second contour, said first andseconds contours being constrained by the aortic valve annulus landmarksand the aortic valve commissure plane landmarks.
 3. The method of claim3, further comprising detecting said global position, anatomicallandmarks, and first and second contours are using marginal spacelearning with a hierarchical approach, wherein detectors aresuccessively trained using probabilistic boosting trees.
 4. The methodof claim 1, further comprising finding an optimal imaging angle for theUS probe by rotating the US probe about its axis, and detecting anangulation of an US fan with respect to the aortic root, and selecting aprobe orientation that maximizes the angulation of the US fan withrespect to the aortic root as the optimal imaging angle.
 5. The methodof claim 1, further comprising inserting the US image into thefluoroscopic image.
 6. The method of claim 1, wherein detecting a poseof the US probe in the 2D cardiac fluoroscopic image comprises:determining a position (u,v), orientation (θy), and size (s) of anultrasound (US) probe in a fluoroscopic image; determining a roll andpitch of the US probe in the fluoroscopic image, wherein the position,orientation, size, roll and pitch comprise pose parameters of the probe;and using said probe pose parameters to transform points in said 2Dcardiac ultrasound image into said 2D cardiac fluoroscopic image,wherein said 2D cardiac ultrasound image is visualized in said 2Dcardiac fluoroscopic image.
 7. The method of claim 6, whereindetermining the position, orientation, and size of the US probe in thefluoroscopic image comprises sequentially applying a classifier for eachof the position, orientation, and size, respectively, wherein eachclassifier is trained using a probabilistic boosting tree.
 8. The methodof claim 7, wherein each of said classifiers is trained using Haar-likefeatures.
 9. The method of claim 7, wherein determining the position ofthe US probe comprises applying a steerable filter to the 2Dfluoroscopic image to identify regions of high contrast which are likelyto contain the US probe.
 10. The method of claim 7, wherein determiningthe size of the US probe comprises detecting two points where a tip ofthe probe meets a shaft of the probe, wherein the orientation andposition of the US probe are used to constrain a search area for thesize detector.
 11. The method of claim 6, wherein determining the rolland pitch of the US probe in the fluoroscopic image comprises matchingan image patch of said fluoroscopic image containing said US probe witheach of a plurality of image templates, wherein each image template isassociated with a particular combination of roll and pitch values,wherein the pitch and roll of a template that best matches the imagepatch are selected as the roll and pitch of the US probe.
 12. A methodof transforming target structure anatomies in a pre-operative image I₂into an intra-operative image I₁, comprising the steps of: determining atransformation Φ aligns a target structure T₂ and an anchor structure A₂in the pre-operative image I2 into a corresponding target structure T₁and anchor structure A₁ in the intra-operative image I₁ by finding atransformation {circumflex over (Φ)} that maximizes a functionallog(P(Φ|I₁, A₂)) using an expectation-maximization approach, wherein thetarget structure T₁ is not visible in the intra-operative image.
 13. Themethod of claim 12, wherein said transformation Φ is a rigidtransformation, wherein an initial transformation Φ⁰ is approximated asa translation, wherein Φ⁰ represents a translation between a barycentera₂ of the anchor anatomy A₂ in the pre-operative image I₂ and a detectedbarycenter a₁ of the anchor anatomy A₁ in the intra-operative image I₁.14. The method of claim 13, wherein initial transformation Φ⁰ isdetermined by a position detector trained by a probabilistic boostingtree classifier and Haar features on the barycenter a₁ of the anchoranatomy A₁ in the intra-operative image I₁.
 15. The method of claim 12,wherein the pericardium is used as the anchor anatomy A₁ and A₂ and theaortic valve is used as the target anatomy T₁ and T₂.
 16. The method ofclaim 12, wherein finding a transformation {circumflex over (Φ)} thatmaximizes a functional log(P(Φ|I₁, A₂)) comprises: generating K sampleΦ_(i) ^(t) point sets (x₁, x₂, x₃, . . . , x_(x)) from the pre-operativeanchor anatomy A₂, wherein each point set comprises N points and eachsample is represented as an isotropic 6D Gaussian distribution Φ_(i)^(t)=N₆(μ_(i),Σ_(i)), Σ_(i)=σ_(i)I, wherein I is an identity matrix andσ_(i) is a one dimensional variable calculated as a kernel function froma probability map F(I) evaluated at the point locations y_(i,j), i=1, .. . , K, j=1, . . . , N; transforming the point sets Φ_(i) ^(t) into theintra-operative image I₁ locations y_(i)*=Φ^(t)(x_(i)), i=1, . . . , K,according to an appearance of the intra-operative image assigning eachpoint y_(i,j)*, j=1, . . . , N from the point set x_(i) to a newlocation y_(i,j), j=1, . . . , N, based on a local appearance of theintra-operative image I₁; approximating final parameters of each sampleΦ_(i) ^(t) by an isotropic Gaussian distribution, wherein a mean μ iscomputed from a least squares solution between the point set Φ_(i) ^(t)in the pre-operative I₂ and the updated point set (y₁, y₂, y₃, . . . ,y_(K)) in the intra-operative image I₁ by minimizing the mapping errorfunction${e_{i} = {\frac{1}{N}{\sum\limits_{j = 1}^{N}{{{\Phi_{i}^{t}( x_{i,j} )} - y_{i,j}}}}}};$and determining an updated global transformation Φ^(t+1) fromΦ^(t+1)=arg_(Φ)max(Φ|Φ^(t)) based on an estimated mixture model$\oplus {= {\sum\limits_{i = 1}^{K}\Phi_{i}^{t}}}$ of the Ktransformation samples Φ_(i) ^(t), i=1, . . . , K .
 17. The method ofclaim 16, wherein$\Phi^{t + 1} = {\underset{\Phi}{\arg \; \max}( \Phi \middle| \Phi^{t} )}$is estimated using a mean shift algorithm.
 18. The method of claim 16,further comprising deriving the probability map F(I₁) from theintra-operative image I₁ by evaluating a boosting classifier trainedusing Haar features and surface annotations of the anchor anatomy A₁ inthe intra-operative image I₁, wherein each vertex of a model of theintra-operative image I₁ is assigned as a positive sample and randompoints within a threshold distance are used as negative in samples, andthose vertices for which a feature response is low are rejected aspositive examples.
 19. The method of claim 18, wherein minimizing saidmapping error further comprises estimating a prior probability for eachvertex of a model of pre-operative image I₂ by assigning each vertex ofa model of the pre-operative image I₂ as a positive sample and usingrandom points within a threshold distance as negative samples, rejectingthose vertices for which a feature response is low as positive examples,estimating a ground-truth mapping Φ_(T) based on hinges and commissuresof the aortic valve, and transforming each intra-operative model of thepre-operative anchor anatomy T₂ into the pre-operative image I₁ usingT₁*=Φ_(T)T₂ and the variance of a point-wise distance ∥T₁*−T₁∥.
 20. Anon-transitory program storage device readable by a computer, tangiblyembodying a program of instructions executed by the computer to performthe method steps for transforming target structure anatomies in apre-operative image I₂ into an intra-operative image I₁, the methodcomprising the steps of: determining a transformation Φ aligns a targetstructure T₂ and an anchor structure A₂ in the pre-operative image I2into a corresponding target structure T₁ and anchor structure A₁ in theintra-operative image I₁ by finding a transformation {circumflex over(Φ)} that maximizes a functional log(P(Φ|I₁, A₂)) using anexpectation-maximization approach, wherein the target structure T₁ isnot visible in the intra-operative image.
 21. The computer readableprogram storage device of claim 20, wherein said transformation Φ is arigid transformation, wherein an initial transformation Φ⁰ isapproximated as a translation, wherein Φ⁰ represents a translationbetween a barycenter a₂ of the anchor anatomy A₂ in the pre-operativeimage I₂ and a detected barycenter a₁ of the anchor anatomy A₁ in theintra-operative image I₁.
 22. The computer readable program storagedevice of claim 21, wherein initial transformation Φ⁰ is determined by aposition detector trained by a probabilistic boosting tree classifierand Haar features on the barycenter a₁ of the anchor anatomy A₁ in theintra-operative image I₁.
 23. The computer readable program storagedevice of claim 20, wherein the pericardium is used as the anchoranatomy A₁ and A₂ and the aortic valve is used as the target anatomy T₁and T₂.
 24. The computer readable program storage device of claim 20,wherein finding a transformation {circumflex over (Φ)} that maximizes afunctional log(P(Φ|I₁, A₂)) comprises: generating K sample Φ_(i) ^(t)point sets (x₁, X₂, X₃, . . . , X_(K)) from the pre-operative anchoranatomy A₂, wherein each point set comprises N points and each sample isrepresented as an isotropic 6D Gaussian distribution Φ_(i)^(t)=N₆(μ_(i),Σ_(i)), Σ_(i)=σ_(i)I, wherein I is an identity matrix andσ_(i) is a one dimensional variable calculated as a kernel function froma probability map F(I) evaluated at the point locations y_(i,j), i=1, .. . , K, j=1, . . . , N; transforming the point sets Φ_(i) ^(t) theintra-operative image I₁ locations y_(i)*=(x_(i)), i=1, . . . , K,according to an appearance of the intra-operative image assigning eachpoint y_(i,j)*, j=1, . . . , N from the point set x_(i) to a newlocation y_(i,j), j=1, . . . , N, based on a local appearance of theintra-operative image I₁; approximating final parameters of each sampleΦ_(i) ^(t) by an isotropic Gaussian distribution, wherein a mean μ iscomputed from a least squares solution between the point set Φ_(i) ^(t)in the pre-operative I₂ and the updated point set (y₁, y₂, y₃, . . . ,y_(K)) in the intra-operative image I₁ by minimizing the mapping errorfunction${e_{i} = {\frac{1}{N}{\sum\limits_{j = 1}^{N}{{{\Phi_{i}^{t}( x_{i,j} )} - y_{i,j}}}}}};$and determining an updated global transformation Φ^(t+1) from$\Phi^{t + 1} = {\underset{\Phi}{\arg \; \max}( \Phi \middle| \Phi^{t} )}$based on an estimated mixture model$\oplus {= {\sum\limits_{i = 1}^{K}\Phi_{i}^{t}}}$ of the Ktransformation samples Φ_(i) ^(t), i=1, . . . , K.
 25. The computerreadable program storage device of claim 24, wherein$\Phi^{t + 1} = {\underset{\Phi}{\arg \; \max}( \Phi \middle| \Phi^{t} )}$is estimated using a mean shift algorithm.
 26. The computer readableprogram storage device of claim 24, the method further comprisingderiving the probability map F(I₁) from the intra-operative image I₁ byevaluating a boosting classifier trained using Haar features and surfaceannotations of the anchor anatomy A₁ in the intra-operative image I₁,wherein each vertex of a model of the intra-operative image I₁ isassigned as a positive sample and random points within a thresholddistance are used as negative samples, and those vertices for which afeature response is low are rejected as positive examples.
 27. Thecomputer readable program storage device of claim 26, wherein minimizingsaid mapping error further comprises estimating a prior probability foreach vertex of a model of pre-operative image I₂ by assigning eachvertex of a model of the pre-operative image I₂ as a positive sample andusing random points within a threshold distance as negative samples,rejecting those vertices for which a feature response is low as positiveexamples, estimating a ground-truth mapping Φ_(T) based on hinges andcommissures of the aortic valve, and transforming each intra-operativemodel of the pre-operative anchor anatomy T₂ into the pre-operativeimage I₁ using T₁*=Φ_(T)T₂ and the variance of a point-wise distance∥T₁*−T₁∥.